A. Field of the Invention
Systems and methods consistent with the principles of the invention relate generally to document processing and, more particularly, to comparing documents to find near duplicate documents.
B. Description of Related Art
There are a number of applications in which it may be desirable to determine whether documents are near duplicates of one another. In the context of the World Wide Web, for example, search engines typically provide a searchable index of numerous web pages. Frequently, web pages located at different locations may be duplicates or near duplicates of one another. Knowing when one web page is a near-duplicate of another can be beneficial both when archiving the web pages and when returning search results to a user in response to a search query.
An archive server, for example, may be designed to store an archive of all documents requested from a web server. The archive server may decide whether to store new incoming documents based on whether the new document is a duplicate or near-duplicate of a previously stored document.
Thus, there is a need in the art for accurate and efficient techniques for automatically detecting near-duplicate documents.